Study-level multi-view processing system

ABSTRACT

One or more processors may identify a missing image in the set of multi-view images. Each of the images is associated with a particular view type. The one or more processors may generate, utilizing a replacement AI model, a replacement image for the missing image in the set of multi-view images. The replacement image is generated utilizing an AI model trained to generate a replacement image using training images from two or more time-adjacent sets of images. The one or more processors may identify a duplicate image in the set of multi-view images. The one or more processors may generate, utilizing a quality generative AI model, a characteristic improved image based on the duplicate image for the set of multi-view images. The one or more processors may output the replacement image and the characteristic improved image.

BACKGROUND

The present disclosure relates generally to the field of medicalimaging, and more specifically to processing images in a set ofmulti-view images by utilizing artificial intelligence models.

Use of artificial intelligence in the field of medical imaging isincreasingly popular. When medical images are utilized to trainartificial intelligence models, the medical images may need to beprocessed to meet the needs of the artificial intelligence models.Occasionally, mammograms obtained from screening exams may have missingor duplicate images. Duplicate images may be taken during screeningexams to address quality issues or due to the limited field of view ofsome images.

SUMMARY

Embodiments of the present disclosure include a method, computer programproduct, and system for processing images in a set of multi-view imagesby utilizing artificial intelligence models.

In some embodiments, one or more processors may identify a missing imagein the set of multi-view images. In some embodiments, each of the imagesin the multi-view set of images is associated with a particular viewtype. In some embodiments, the one or more processors may generate,utilizing a replacement AI model, a replacement image for the missingimage in the set of multi-view images. In some embodiments, thereplacement image is generated utilizing an AI model trained to generatea replacement image using training images from two or more time-adjacentsets of images. In some embodiments, the one or more processors mayidentify a duplicate image in the set of multi-view images. In someembodiments, the duplicate image is an image of the same view type asanother image in the set of multi-view images. In some embodiments, theone or more processors may generate, utilizing a quality generative AImodel, a characteristic improved image based on the duplicate image forthe set of multi-view images. In some embodiments, the one or moreprocessors may output the replacement image and the characteristicimproved image.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings included in the present disclosure are incorporated into,and form part of, the specification. They illustrate embodiments of thepresent disclosure and, along with the description, serve to explain theprinciples of the disclosure. The drawings are only illustrative ofcertain embodiments and do not limit the disclosure.

FIG. 1 is a block diagram of an exemplary system for processing imagesin a set of multi-view images, in accordance with aspects of the presentdisclosure.

FIG. 2 is a flowchart of an exemplary method for processing images in aset of multi-view images, in accordance with aspects of the presentdisclosure.

FIG. 3A illustrates a cloud computing environment, in accordance withaspects of the present disclosure.

FIG. 3B illustrates abstraction model layers, in accordance with aspectsof the present disclosure.

FIG. 4 illustrates a high-level block diagram of an example computersystem that may be used in implementing one or more of the methods,tools, and modules, and any related functions, described herein, inaccordance with aspects of the present disclosure.

While the embodiments described herein are amenable to variousmodifications and alternative forms, specifics thereof have been shownby way of example in the drawings and will be described in detail. Itshould be understood, however, that the particular embodiments describedare not to be taken in a limiting sense. On the contrary, the intentionis to cover all modifications, equivalents, and alternatives fallingwithin the spirit and scope of the disclosure.

DETAILED DESCRIPTION

Aspects of the present disclosure relate generally to the field ofmedical imaging, and more specifically to processing images in a set ofmulti-view images by utilizing artificial intelligence models. While thepresent disclosure is not necessarily limited to such applications,various aspects of the disclosure may be appreciated through adiscussion of various examples using this context.

Use of artificial intelligence (“AI”) in the field of medical imaging isincreasingly popular. When medical images are utilized to trainartificial intelligence models, the medical images may need to beprocessed to meet the needs of the artificial intelligence models.Occasionally, mammograms obtained from screening exams may have missingor duplicate images. Duplicate images may be taken during screeningexams to address quality issues, due to a limited field of view of someimages, or where some images are magnified views.

Typically, images that are used to train AI models are in sets ofspecific numbers of images, with each image having a particular viewtype. When there are duplicate images it can be difficult for a user toselect which of the training images to provide to the AI. A link betweenprior and current exams may not be straight-forward when there aremultiple valid images (e.g., multiple high-quality images of the sameobject, etc.). Further, Study-level Computer Aided Detection (“CAD”)design is limited by sets of images with missing or duplicate images.Therefore, there is a need in the field of medical imaging for a method,which is disclosed herein, that provides a manner to manage missingand/or duplicate images.

In some embodiments, a method for processing images in a set ofmulti-view images utilizing AI is provided. In some embodiments, one ormore processors may identify a missing image in the set of multi-viewimages. In some embodiment, each of the images in the set of multi-viewimages may be associated with a particular view type. In someembodiments, the set of multi-view images may include, when not missingan image, images of a predetermined variety of view types. In someembodiments, the multi-view set of images may be a set of mammogramimages.

For example, the set of mammograms images may include images of fourview types, left CC view, right CC view, left MLO view, and right MLOview. Each of the views may be associated with a type of image capturedduring a mammogram screening exam (e.g., having varied spatialorientations from where the images were taken) and a particular organ(e.g., right or left).

In some embodiments, the missing image may be identified based onidentification data that is associated with, and identifies, the viewtype of each image in the set. In some embodiments, the identificationdata may be extracted by the one or more processors during preprocessingfrom data sets associated with the image. In some embodiments, the datasets may include information obtained from user input (e.g.,annotations) or from reports (e.g., medical reports, computer aideddiagnosis (“CAD”) marks) associated with the images.

In some embodiments, the images in the set of multi-view images may bemedical images that reveal internal structures of the body hidden by theskin. The images may be radiological images generated through imagingtechnologies, such as X-ray radiography, magnetic resonance imaging,ultrasound, endoscopy, elastography, tactile imaging, thermography,medical photography, and/or nuclear medicine functional imagingtechniques, such as, positron emission tomography (PET) andsingle-photon emission computed tomography (SPECT). In some embodiments,the images may be digital X-rays. In some embodiments, the images may bemammograms.

In some embodiments, the one or more processors may generate, utilizinga replacement AI model, a replacement image for the missing image in theset of multi-view images. In some embodiments, the replacement image maybe generated utilizing an AI model trained to generate a replacementimage using training images from two or more time-adjacent sets ofimages. In some embodiments, the replacement image may be generated tobe of the same view type as the missing image.

For example, if the missing image is a left CC view image, thereplacement image generated by the replacement AI model may be a left CCview image. The replacement AI model may be trained to generate areplacement image using images of the same view type (e.g., left CCview) from time-adjacent images of the same view type. In someembodiments, the time-adjacent images of the same view type may beobtained from time-adjacent sets of images where at least two of thetime-adjacent sets of images are not missing images of the particularview type (e.g., left CC view).

For example, each set of images may be a set of mammogram images havingimages of various view types taken during periodic (e.g., yearly,bi-annual, etc.) screening exams, and the time-adjacent sets of imagesmay be mammogram images taken during screening exams, which directlypreceded or followed the screening exam during which the set ofmammogram images for which a replacement image is generated was taken.In some embodiments, the replacement AI model utilizes a generativeadversarial network architecture.

In some embodiments, the one or more processors may identify a duplicateimage in the set of multi-view images. In some embodiments, theduplicate image may be an image of the same view type as another imagein the set of multi-view images. For example, the one or more processorsmay identify, based on identification data associated with each image,that there are two images labeled as right CC view images in the set ofmulti-view images.

In some embodiments, the one or more processors, may generate, utilizinga quality generative AI model, a characteristic improved image based onthe duplicate image for the set of multi-view images. For example, apair of images that are duplicates (e.g., two right CC view images) maybe input into a quality generative AI model. The quality generative AImodel may output an image of the right CC view type, where the outputright CC view type image is the characteristic improved image. In someembodiments, the characteristic improved image may have at least oneimage characteristic, which is equal to or better than that of one (orboth) of the input images. In some embodiments, the image characteristicmay be a characteristic indicative of the quality of the image (e.g.,resolution, sharpness or blur, motion blurring, characteristicsindicative of geometric magnification, characteristics indicative ofspot or localized compression, characteristics indicative of use of acontrast agent, or characteristics indicative of dual energysubtraction, etc.). In some embodiments, the image characteristic may bedegraded manually using known image manipulation techniques (e.g.,making the image blurrier, etc.) to create training images to train themodel.

In some embodiments, the one or more processors may output thereplacement image and the characteristic improved image. In someembodiments, the replacement image and the characteristic improved imagemay be output (e.g., displayed) to a user to view. In some embodiments,the characteristic improved image and the replacement image may beoutput to a computer system that utilizes the set of multi-view images,which have a predetermined number of images having a predeterminedvariety of view types, as data for training machine learning models.

For example, the set of multi-view images may be a set of mammogramimages taken during a screening exam for a patient having four images,one of each of the following types: right CC, left CC, right MLO, leftMLO. The CC& MLO views are standard screening views but others may beused or added to aid in visualization of lesions or differentiatebetween structures that may be superimposed in one of the views. Theseimages may be utilized to perform studies related to the treatment ordiagnosis of breast cancer, developing machine learning techniques toaid in the treatment or diagnosis of breast cancer, or developingmachine learning techniques related to image classification, objectdetection, lesion classification (e.g., calcification, solid mass, cyst,benign, malignant, suspicious, or probably malignant), image qualitycontrol (e.g., where some images or the study may be rejected to bere-acquired due to technical defects such as poor positioning/lack ofrequired anatomical coverage, motion blurring, etc.), etc.

In some embodiments, the one or more processors may identify a partialimage and a second image, where the partial image and the second imageare of the same view type as each other. For example, the partial imageand the second image in the set of multi-view images may both be labeledas right MLO images. In some embodiments, the partial image and thesecond image may be identified based on identification data that isassociated with, and identifies, the view type of each image in the set.

In some embodiments, the partial image may be detected based on a pixelby pixel comparison of a feature in the image. In some embodiments,whether a feature of the pixels exceeds a threshold may be determined,and each pixel in the image may be compared to a neighboring pixel inthe image. For example, a pixel in the image may exceed a threshold,indicating that the pixel is of an opaque background. Neighboring pixelsmay also be of an opaque background. A certain number of opaquebackground pixels may indicate that the area in the image covered inopaque background pixels does not show a portion of the organ, and theimage of the organ may not have been fully captured. Furthering theexample, a certain number of opaque background pixels may encompass anarea of non-opaque pixels and the opaque background pixels may borderthe edges of the image, which may indicate that the image is a fullimage.

In some embodiments, the one or more processors may generate, utilizinga stitch AI model, a combined image having a combined field of view ofthe partial image and the second image. For example, the partial imagemay be a medical image of an organ that shows the right most ⅔ of theorgan, and the second image may be a partial medical image of the organwhich shows the left most ⅔ of the organ. The combined field of viewimage generated utilizing the stitch AI model may show the entire organincluding the right and left portions of the organ and the centerportion showing ⅓ of the organ that is shown in both the partial imageand the second image.

As another example, the partial image may be a zoomed-in (e.g.,magnified) medical image of a small portion of an organ, and the secondimage may be a complete image that shows the entire organ (e.g., azoomed-out/global view). In some embodiments, the magnified view imagemay not be a simple zoomed in view, but rather a view obtained from adifferent x-ray capture technique that utilizes geometric magnificationof the organ. The magnified image may have higher resolution in theplane of the organ but also a limited field of view. The combined fieldof view image generated utilizing the stitch AI model may show theentire organ including the zoomed-in portion of the organ shown in thepartial image.

In some embodiments, there may be multiple second images and the stitchAI model may be applied, in succession, to pairs of images from the setincluding the partial image and the multiple second images until asingle final image is produced. For example, the stitch AI model maycombine three images together by first combing two of the three imagestogether to generate a new image and then combining the new image andthe third image together. In some embodiments, the stitch AI model maybe a GAN.

In some embodiments, the stitch AI model may be trained by utilizing alocal discriminator to stitch high resolution boundaries between the oneor more partial images and utilizing a global discriminator to ensurefull-image cohesiveness. The local discriminator may also preserve localdiagnostic regions as in the original image. For example, one of thepartial images may be a zoomed-in portion of the organ (for example, adiagnostically important area that was identified in annotationsassociated with the image) that is combined with an image of the entireorgan using the stitch AI model. In some embodiments, the localdiscriminator may preserve the higher magnification and higher detailfrom the zoomed-in portion as well as the original, or allow themagnification and detail from the zoomed-in portion to be greater thanthe magnification and detail in the image of the entire organ, and allowfor realistic detail at the boundary.

In some embodiments, during training of the replacement AI model, theone or more processors may receive three training sets of images from atraining user, where each image in each training set of images isassociated with a particular view type of a set of view types. In someembodiments, the one or more processors may identify that one of thethree training sets of images is an incomplete training set. In someembodiments, the incomplete training set may have a missing image. Forexample, an image may be missing from the incomplete training set whenthere are no images of a particular view type among the images in theset.

In some embodiments, the one or more processors may identify twotime-adjacent training sets. The two time-adjacent training sets may besets of medical images taken during different screening exams (e.g.,taken at different times). The two time-adjacent training sets may befrom screening exams that immediately preceded and immediately followed,respectively, a screening exam during which the medical images in theincomplete training set were taken. In some embodiments, the incompletetraining set may be created by removing an image from a set of imagesthat was previously not incomplete (e.g., complete).

In some embodiments, from each of the two time-adjacent training sets,an image of the view type of the missing image may be obtained and inputto the replacement AI model. A generator may then be utilized to fill inthe missing image based on the images of the view type of the missingimage from the two-time adjacent training sets. In some embodiments, thereplacement AI model may have a generative adversarial networkarchitecture. During training of the replacement AI model, adiscriminator may be used to evaluate the performance of the generatorin generating a replacement image. The discriminator may ensure that thegenerated images appear realistic in the context of the correspondingtime-adjacent images and related anatomy from the current study (e.g.,the set of images having a missing image). In some embodiments, thediscriminator utilizes a standard loss function.

An exemplary method for generating a replacement image may involveinputting into a replacement AI model an all-black (e.g., all-opaque)image in place of the missing image. The generator may then fill in theall-black image based on the images of the view type of the missingimage from the time-adjacent training sets. The missing regions may befilled in using cues from the other images in the same study and theimages in the time-adjacent studies. For example, the left MLO may bemissing in the current study (e.g., set of images having a missingimage) and present in the prior study (e.g., a set of time-adjacentimages). There may be a lesion visible in the current left CC image thatis larger than in the prior study. The generator may use the generalanatomical structure from the prior left MLO combined with the lesiongrowth information from the current left CC image. In some embodiments,if there is more than one missing image, all missing images in themulti-view set of images may be filled in by repeatedly applying thismethod to each pair of images from time-adjacent training sets.

In some embodiments, during training of the replacement AI model, one ormore processors may receive four or more training sets of images from atraining user, where each image in each training set of images isassociated with a particular view type of a set of view types. In someembodiments, the one or more processors may identify that one of thefour or more training sets is an incomplete training set. The three ormore time-adjacent training sets may be from screening exams that eachpreceded or followed a screening exam during what time the medicalimages in the incomplete training set were taken.

From each of the three or more time-adjacent training sets, an image ofthe view type of the missing image may be obtained and input to thereplacement AI model. A time-series generative model may then beutilized to fill in the missing image based on the images of the viewtype of the missing image from the three or more time-adjacent trainingsets. In some embodiments, the time-series generative model may have arecurrent neural network (“RNN”) architecture, a temporal convolutionalnetwork (“TCN”) architecture, a long short-term memory (“LSTM”)architecture, or any other architecture for a time-series generativemodel. The timeseries generative model may allow more than twotime-adjacent images to be input to generate the replacement image. Insome embodiments, the replacement AI model may have a generativeadversarial network architecture. During training of the replacement AImodel, a discriminator may be used to evaluate the performance of thegenerator in generating a replacement image.

In some embodiments, an exemplary method for generating a replacementimage may involve the generator receiving an arbitrary number of inputstudies (e.g., sets of images collected from a screening exam) as a timeseries. Stacked images from a study may then be reshaped intoone-dimensional vectors and the generator model becomes a temporalconvolutional network. An all-black image is input into a replacement AImodel in place of the missing image. The replacement image isconstructed by reshaping the one-dimensional vector back into atwo-dimensional image.

In some embodiments, during training of the quality generative AI model,one or more processors may identify two or more training images from auser, where the two or more training images are of the same view type.In some embodiments, the two or more training images of the same viewtype may be identified by identification data. In some embodiments, thetwo or more training images may be input into a quality generative AImodel. In some embodiments, a characteristic improved image may begenerated, utilizing the quality generative AI model, based on the twoor more training images of the same view type. In some embodiments, thequality generative AI model may be any machine learning model which isconfigured to be utilized to generate images based on two or more imagesof a similar type. In some embodiments, the quality generative AI modelmay utilize a GAN architecture.

In some embodiments, during training of the quality generative AI model,a global discriminator may be used to train the quality generative AImodel to generate a realistic image and/or balance between highresolution and quality. For example, two images of the same view typemay be two images of the entire organ where one image shows some tissuesurrounding the organ (e.g., the pectoral muscle). The characteristicimproved image may be generated to show the tissue surrounding theorgan. As another example, one of the two images of the same view typemay be out of focus. The characteristic improved image may be generatedto have greater focus than the out of focus image.

In some embodiments, during training of the replacement AI model, alocal discriminator may be used to train the generator to synthesizecharacteristic improved images with preserved local diagnostic regions.For example, two images of the same view type may include an image of amagnified view of a portion of the organ. The local discriminator may beused to train the generator to preserve the greater magnification of theportion of the organ, and the global discriminator may be used to trainthe generator to generate realistic images which transition from themagnified portion to the remainder of the organ without unrealisticlines or boundaries between regions of the organ.

In some embodiments, a quality map may be generated, utilizing a qualityassessment AI model, for a training image of the training images havingthe same view type based on detecting one or more low quality areas. Thequality map may be a representation of the training image thatidentifies one or more low quality areas. In some embodiments, thequality map may be a heat map of the quality rating for each pixel ofthe training image. For example, the quality map may be a pixel by pixelmap that provides a quality rating (e.g., 0 to 1 numeric score) to eachpixel of the training image, and a quality map may be generated for eachof the two or more training images having the same view type.

In some embodiments, the quality map may be input into the qualitygenerative AI model, where a characteristic improved image is generatedbased on the quality map input into the quality generative AI model. Forexample, for each of the training images of the same view types, aquality map may be input along with the training image into the qualitygenerative AI model. A characteristic improved image may then begenerated based on the quality maps provided. The one or more qualitymaps may guide the quality generative AI model to generate higherquality images.

In some embodiments, the quality generative AI model utilizes atime-series generative model. For example, when there are three (orpossibly more) training images of the same view type, all three trainingimages may be combined by the AI model to generate a characteristicimproved image by utilizing a generator that is a time-series generativemodel. The time series generative model may treat the multiple images asa time series and allow the quality generative AI model to take anarbitrary number of inputs to generate an arbitrary number of outputs.

Referring now to FIG. 1, a block diagram of a network 100 for processingimages in a set of multi-view images utilizing artificial intelligenceis illustrated. Network 100 includes a user device 102 and an AI system104, which are configured to be in communication with each other. Insome embodiments, first device 102 may be any device that contains aprocessor configured to perform one or more of the functions, or steps,described herein this disclosure. AI system 104 includes a replacementAI model 106, a quality generative AI model 110, a stitch AI model 112,and a quality assessment AI model 112.

In some embodiments, AI system 104 identifies a missing image in the setof multi-view images. In some embodiments, each of the images in themulti-view set of images is associated with a particular view type. Insome embodiments, AI system 104 generates, utilizing the replacement AImodel 106, a replacement image for the missing image in the set ofmulti-view images.

In some embodiments, the replacement image is generated utilizing an AImodel trained to generate a replacement image using training images fromtwo or more time-adjacent sets of images. In some embodiments, AI system104 identifies a duplicate image in the set of multi-view images. Insome embodiments, the duplicate image is an image of the same view typeas another image in the set of multi-view images. In some embodiments,AI system 104 generates, utilizing the quality generative AI model 108,a characteristic improved image based on the duplicate image for the setof multi-view images.

In some embodiments, AI system 104 generates, utilizing the qualityassessment AI model 112, a quality map for the training images havingthe same view type based on detecting one or more low quality areas. Insome embodiments, the quality map is input into the quality generativeAI model 108, where the characteristic improved image is generated basedon the quality map input into the quality generative AI model 108. Insome embodiments, AI system 104 identifies a partial image and a secondimage. In some embodiments, the partial image and the second image areof the same view type as each other. In some embodiments, AI system 104generates, utilizing a stitch AI model 110, a combined image having acombined field of view of the partial image and the second image.

In some embodiments, AI system 104 outputs the replacement image, thecharacteristic improved image, and the combined image. In someembodiments, the user device 102 has a user interface by which a usermay view the replacement image, the characteristic improved image, orthe combined image. In some embodiments, the user device 102 is part ofa computer system having one or more computing devices which utilize theset of multi-view images as data for training machine learning models.

Referring now to FIG. 2, illustrated is a flowchart of an exemplarymethod 200 for processing images in a set of multi-view images utilizingartificial intelligence, in accordance with aspects of the presentdisclosure. In some embodiments, one or more processors of a system(e.g., an AI system) may perform the operations of the method 200. Insome embodiments, method 200 begins at operation 202. At operation 202,the AI system identifies a missing image in the set of multi-viewimages. In some embodiments, each of the images in the multi-view set ofimages is associated with a particular view type.

In some embodiments, method 200 proceeds to operation 204, where the AIsystem generates, utilizing a replacement AI model, a replacement imagefor the missing image in the set of multi-view images. In someembodiments, the replacement image is generated utilizing an AI modeltrained to generate a replacement image using training images from twoor more time-adjacent sets of images. In some embodiments, method 200proceeds to operation 206. At operation 206, the AI system identifies aduplicate image in the set of multi-view images. In some embodiments,the duplicate image is an image of the same view type as another imagein the set of multi-view images.

In some embodiments, method 200 proceeds to operation 208. At operation208, the AI system generates, utilizing a quality generative AI model, acharacteristic improved image based on the duplicate image for the setof multi-view images. In some embodiments, method 200 proceeds tooperation 210. At operation 210, the AI system outputs the replacementimage and the characteristic improved image. In some embodiments, afteroperation 210 method 200 may end.

As discussed in more detail herein, it is contemplated that some or allof the operations of the method 200 may be performed in alternativeorders or may not be performed at all; furthermore, multiple operationsmay occur at the same time or as an internal part of a larger process.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present disclosure are capable of being implementedin conjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of portion independence in that the consumergenerally has no control or knowledge over the exact portion of theprovided resources but may be able to specify portion at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based email). Theconsumer does not manage or control the underlying cloud infrastructureincluding network, servers, operating systems, storage, or evenindividual application capabilities, with the possible exception oflimited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

FIG. 3A, illustrated is a cloud computing environment 310 is depicted.As shown, cloud computing environment 310 includes one or more cloudcomputing nodes 300 with which local computing devices used by cloudconsumers, such as, for example, personal digital assistant (PDA) orcellular telephone 300A, desktop computer 300B, laptop computer 300C,and/or automobile computer system 300N may communicate. Nodes 300 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof.

This allows cloud computing environment 310 to offer infrastructure,platforms and/or software as services for which a cloud consumer doesnot need to maintain resources on a local computing device. It isunderstood that the types of computing devices 300A-N shown in FIG. 3Aare intended to be illustrative only and that computing nodes 300 andcloud computing environment 310 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

FIG. 3B, illustrated is a set of functional abstraction layers providedby cloud computing environment 310 (FIG. 3A) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 3B are intended to be illustrative only and embodiments of thedisclosure are not limited thereto. As depicted below, the followinglayers and corresponding functions are provided.

Hardware and software layer 315 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 302;RISC (Reduced Instruction Set Computer) architecture based servers 304;servers 306; blade servers 308; storage devices 311; and networks andnetworking components 312. In some embodiments, software componentsinclude network application server software 314 and database software316.

Virtualization layer 320 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers322; virtual storage 324; virtual networks 326, including virtualprivate networks; virtual applications and operating systems 328; andvirtual clients 330.

In one example, management layer 340 may provide the functions describedbelow. Resource provisioning 342 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 344provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 346 provides access to the cloud computing environment forconsumers and system administrators. Service level management 348provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 350 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 360 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 362; software development and lifecycle management 364;virtual classroom education delivery 366; data analytics processing 368;transaction processing 370; and processing images in a set of multi-viewimages 372.

FIG. 4, illustrated is a high-level block diagram of an example computersystem 401 that may be used in implementing one or more of the methods,tools, and modules, and any related functions, described herein (e.g.,using one or more processor circuits or computer processors of thecomputer), in accordance with embodiments of the present disclosure. Insome embodiments, the major components of the computer system 401 maycomprise one or more CPUs 402, a memory subsystem 404, a terminalinterface 412, a storage interface 416, an I/O (Input/Output) deviceinterface 414, and a network interface 418, all of which may becommunicatively coupled, directly or indirectly, for inter-componentcommunication via a memory bus 403, an I/O bus 408, and an I/O businterface unit 410.

The computer system 401 may contain one or more general-purposeprogrammable central processing units (CPUs) 402A, 402B, 402C, and 402D,herein generically referred to as the CPU 402. In some embodiments, thecomputer system 401 may contain multiple processors typical of arelatively large system; however, in other embodiments the computersystem 401 may alternatively be a single CPU system. Each CPU 402 mayexecute instructions stored in the memory subsystem 404 and may includeone or more levels of on-board cache.

System memory 404 may include computer system readable media in the formof volatile memory, such as random access memory (RAM) 422 or cachememory 424. Computer system 401 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 426 can be provided forreading from and writing to a non-removable, non-volatile magneticmedia, such as a “hard drive.” Although not shown, a magnetic disk drivefor reading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), or an optical disk drive for reading from orwriting to a removable, non-volatile optical disc such as a CD-ROM,DVD-ROM or other optical media can be provided. In addition, memory 404can include flash memory, e.g., a flash memory stick drive or a flashdrive. Memory devices can be connected to memory bus 403 by one or moredata media interfaces. The memory 404 may include at least one programproduct having a set (e.g., at least one) of program modules that areconfigured to carry out the functions of various embodiments.

One or more programs/utilities 428, each having at least one set ofprogram modules 430 may be stored in memory 404. The programs/utilities428 may include a hypervisor (also referred to as a virtual machinemonitor), one or more operating systems, one or more applicationprograms, other program modules, and program data. Each of the operatingsystems, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. Programs 428 and/or program modules 430generally perform the functions or methodologies of various embodiments.

Although the memory bus 403 is shown in FIG. 4 as a single bus structureproviding a direct communication path among the CPUs 402, the memorysubsystem 404, and the I/O bus interface 410, the memory bus 403 may, insome embodiments, include multiple different buses or communicationpaths, which may be arranged in any of various forms, such aspoint-to-point links in hierarchical, star or web configurations,multiple hierarchical buses, parallel and redundant paths, or any otherappropriate type of configuration. Furthermore, while the I/O businterface 410 and the I/O bus 408 are shown as single respective units,the computer system 401 may, in some embodiments, contain multiple I/Obus interface units 410, multiple I/O buses 408, or both. Further, whilemultiple I/O interface units are shown, which separate the I/O bus 408from various communications paths running to the various I/O devices, inother embodiments some or all of the I/O devices may be connecteddirectly to one or more system I/O buses.

In some embodiments, the computer system 401 may be a multi-usermainframe computer system, a single-user system, or a server computer orsimilar device that has little or no direct user interface, but receivesrequests from other computer systems (clients). Further, in someembodiments, the computer system 401 may be implemented as a desktopcomputer, portable computer, laptop or notebook computer, tabletcomputer, pocket computer, telephone, smartphone, network switches orrouters, or any other appropriate type of electronic device.

It is noted that FIG. 4 is intended to depict the representative majorcomponents of an exemplary computer system 401. In some embodiments,however, individual components may have greater or lesser complexitythan as represented in FIG. 4, components other than or in addition tothose shown in FIG. 4 may be present, and the number, type, andconfiguration of such components may vary.

As discussed in more detail herein, it is contemplated that some or allof the operations of some of the embodiments of methods described hereinmay be performed in alternative orders or may not be performed at all;furthermore, multiple operations may occur at the same time or as aninternal part of a larger process.

The present disclosure may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present disclosure.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astandalone software package, partly on the user's computer and partly ona remote computer or entirely on the remote computer or server. In thelatter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

Although the present disclosure has been described in terms of specificembodiments, it is anticipated that alterations and modification thereofwill become apparent to the skilled in the art. Therefore, it isintended that the following claims be interpreted as covering all suchalterations and modifications as fall within the true spirit and scopeof the disclosure.

What is claimed is:
 1. A computer-implemented method for processingimages in a set of multi-view images utilizing artificial intelligence(“AI”), the method comprising: identifying, by one or more processors, amissing image in the set of multi-view images, wherein each of theimages in the multi-view set of images is associated with a particularview type; generating, utilizing a replacement AI model, a replacementimage for the missing image in the set of multi-view images, wherein thereplacement image is generated utilizing an AI model trained to generatea replacement image using training images from two or more time-adjacentsets of images; identifying a duplicate image in the set of multi-viewimages, wherein the duplicate image is an image of the same view type asanother image in the set of multi-view images; generating, utilizing aquality generative AI model, a characteristic improved image based onthe duplicate image for the set of multi-view images; and outputting thereplacement image and the characteristic improved image.
 2. The methodof claim 1, further comprising: identifying a partial image and a secondimage, wherein the partial image and the second image are of the sameview type as each other; and generating, utilizing a stitch AI model, acombined image having a combined field of view of the partial image andthe second image.
 3. The method of claim 1, wherein the replacement AImodel was trained by: receiving three training sets of images from atraining user, wherein each of the images in each of the training setsof images is associated with a particular view type of a set of viewtypes; identifying an incomplete training set of the three trainingsets, wherein the incomplete training set has a missing image, whereinthe missing image is associated with a particular view type; identifyingtwo time-adjacent training sets of the three training sets, wherein thetwo time-adjacent training sets are from two exams that aretime-adjacent to an exam of the incomplete training set; and utilizing agenerator to fill in the missing image based on a training image fromeach of the two time-adjacent training sets.
 4. The method of claim 1,wherein the replacement AI model was trained by: retrieving four or moretraining sets of images from a training user, wherein each of the imagesin each of the training sets of images is associated with a particularview type of a set of view types; identifying an incomplete training setof the four or more training sets of images, wherein the incompletetraining set has a missing image; identifying three or moretime-adjacent training sets, wherein the three or more time-adjacenttraining sets are from exams that are time-adjacent to an exam of theincomplete training set; and utilizing a time series generative model tofill in the missing image based on a training image from each of thethree or more time-adjacent training sets.
 5. The method of claim 1,wherein the quality generative AI model was trained by: identifying twoor more training images from a user, wherein the two or more trainingimages are of the same view type; inputting the two or more trainingimages into a quality generative AI model; generating, utilizing thequality generative AI model, a characteristic improved image based onthe two or more training images; and utilizing a global discriminator totrain the quality generative AI model to generate an image.
 6. Themethod of claim 5, wherein the quality generative AI model was furthertrained by: utilizing a local discriminator to train the generator tosynthesize characteristic improved images with preserved localdiagnostic regions.
 7. The method of claim 5, wherein the qualitygenerative AI model was further trained by: generating, utilizing aquality assessment AI model, a quality map for the training imageshaving the same view type based on detecting one or more low qualityareas; and inputting the quality map into the quality generative AImodel, wherein the characteristic improved image is generated based onthe quality map input into the quality generative AI model.
 8. Themethod of claim 1, wherein the stitch AI model was trained by: utilizinga local discriminator to stitch high resolution boundaries between theone or more partial images; and utilizing a global discriminator toensure full image cohesiveness.
 9. A system comprising: a memory; and aprocessor in communication with the memory, the processor beingconfigured to perform operations comprising: identifying a missing imagein the set of multi-view images, wherein each of the images in themulti-view set of images is associated with a particular view type;generating, utilizing a replacement AI model, a replacement image forthe missing image in the set of multi-view images, wherein thereplacement image is generated utilizing an AI model trained to generatea replacement image using training images from two or more time-adjacentsets of images; identifying a duplicate image in the set of multi-viewimages, wherein the duplicate image is an image of the same view type asanother image in the set of multi-view images; generating, utilizing aquality generative AI model, a characteristic improved image based onthe duplicate image for the set of multi-view images; and outputting thereplacement image and the characteristic improved image.
 10. The systemof claim 9, the processor being further configured to perform operationscomprising: identifying a partial image and a second image, wherein thepartial image and the second image are of the same view type as eachother; and generating, utilizing a stitch AI model, a combined imagehaving a combined field of view of the partial image and the secondimage.
 11. The system of claim 9, wherein the replacement AI model wastrained by: receiving three training sets of images from a traininguser, wherein each of the images in each of the training sets of imagesis associated with a particular view type of a set of view types;identifying an incomplete training set of the three training sets,wherein the incomplete training set has a missing image, wherein themissing image is associated with a particular view type; identifying twotime-adjacent training sets of the three training sets, wherein the twotime-adjacent training sets are from two exams that are time-adjacent toan exam of the incomplete training set; and utilizing a generator tofill in the missing image based on a training image from each of the twotime-adjacent training sets.
 12. The system of claim 9, wherein thereplacement AI model was trained by: retrieving four or more trainingsets of images from a training user, wherein each of the images in eachof the training sets of images is associated with a particular view typeof a set of view types; identifying an incomplete training set of thefour or more training sets of images, wherein the incomplete trainingset has a missing image; identifying three or more time-adjacenttraining sets, wherein the three or more time-adjacent training sets arefrom exams that are time-adjacent to an exam of the incomplete trainingset; and utilizing a time-series generative model to fill in the missingimage based on a training image from each of the three or moretime-adjacent training sets.
 13. The system of claim 9, wherein thequality generative AI model was trained by: identifying two or moretraining images from a user, wherein the two or more training images areof the same view type; inputting the two or more training images into aquality generative AI model; generating, utilizing the qualitygenerative AI model, a characteristic improved image based on the two ormore training images; and utilizing a global discriminator to train thequality generative AI model to generate an image.
 14. The system ofclaim 13, wherein the quality generative AI model was further trainedby: utilizing a local discriminator to train the generator to synthesizecharacteristic improved images with preserved local diagnostic regions.15. The system of claim 13, wherein the quality generative AI model wasfurther trained by: generating, utilizing a quality assessment AI model,a quality map for the training images having the same view type based ondetecting one or more low quality areas; and inputting the quality mapinto the quality generative AI model, wherein the characteristicimproved image is generated based on the quality map input into thequality generative AI model.
 16. A computer program product comprising anon-transitory computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya processor to cause the processor to perform operations, the operationscomprising: identifying a missing image in the set of multi-view images,wherein each of the images in the multi-view set of images is associatedwith a particular view type; generating, utilizing a replacement AImodel, a replacement image for the missing image in the set ofmulti-view images, wherein the replacement image is generated utilizingan AI model trained to generate a replacement image using trainingimages from two or more time-adjacent sets of images; identifying aduplicate image in the set of multi-view images, wherein the duplicateimage is an image of the same view type as another image in the set ofmulti-view images; generating, utilizing a quality generative AI model,a characteristic improved image based on the duplicate image for the setof multi-view images; and outputting the replacement image and thecharacteristic improved image.
 17. The computer program product of claim16, the processor being further configured to perform operationscomprising: identifying a partial image and a second image, wherein thepartial image and the second image are of the same view type as eachother; and generating, utilizing a stitch AI model, a combined imagehaving a combined field of view of the partial image and the secondimage.
 18. The computer program product of claim 16, wherein thereplacement AI model was trained by: receiving three training sets ofimages from a training user, wherein each of the images in each of thetraining sets of images is associated with a particular view type of aset of view types; identifying an incomplete training set of the threetraining sets, wherein the incomplete training set has a missing image,wherein the missing image is associated with a particular view type;identifying two time-adjacent training sets of the three training sets,wherein the two time-adjacent training sets are from two exams that aretime-adjacent to an exam of the incomplete training set; and utilizing agenerator to fill in the missing image based on a training image fromeach of the two time-adjacent training sets.
 19. The computer programproduct of claim 16, wherein the quality generative AI model was trainedby: identifying two or more training images from a user, wherein the twoor more training images are of the same view type; inputting the two ormore training images into a quality generative AI model; generating,utilizing the quality generative AI model, a characteristic improvedimage based on the two or more training images; and utilizing a globaldiscriminator to train the quality generative AI model to generate animage.
 20. The computer program product of claim 19, wherein the qualitygenerative AI model was further trained by: generating, utilizing aquality assessment AI model, a quality map for the training imageshaving the same view type based on detecting one or more low qualityareas; and inputting the quality map into the quality generative AImodel, wherein the characteristic improved image is generated based onthe quality map input into the quality generative AI model.